Skip to content

Add support for ARIES UNDO and constant time recovery#28

Draft
gburd wants to merge 11 commits into
masterfrom
undo
Draft

Add support for ARIES UNDO and constant time recovery#28
gburd wants to merge 11 commits into
masterfrom
undo

Conversation

@gburd

@gburd gburd commented Jul 3, 2026

Copy link
Copy Markdown
Owner

No description provided.

@github-actions github-actions Bot force-pushed the master branch 4 times, most recently from 6850c08 to 3b6f413 Compare July 3, 2026 20:55
@github-actions github-actions Bot force-pushed the master branch 10 times, most recently from c3ee91f to c7e7712 Compare July 6, 2026 18:53
@github-actions github-actions Bot force-pushed the master branch 13 times, most recently from c2a5026 to d8fed9e Compare July 8, 2026 08:11
  - Hourly upstream sync from postgres/postgres (24x daily)
  - AI-powered PR reviews using AWS Bedrock Claude Sonnet 4.5
  - Multi-platform CI via existing Cirrus CI configuration
  - Cost tracking and comprehensive documentation

  Features:
  - Automatic issue creation on sync conflicts
  - PostgreSQL-specific code review prompts (C, SQL, docs, build)
  - Cost limits: $15/PR, $200/month
  - Inline PR comments with security/performance labels
  - Skip draft PRs to save costs

  Documentation:
  - .github/SETUP_SUMMARY.md - Quick setup overview
  - .github/QUICKSTART.md - 15-minute setup guide
  - .github/PRE_COMMIT_CHECKLIST.md - Verification checklist
  - .github/docs/ - Detailed guides for sync, AI review, Bedrock

  See .github/README.md for complete overview

Complete Phase 3: Windows builds + fix sync for CI/CD commits

Phase 3: Windows Dependency Build System
- Implement full build workflow (OpenSSL, zlib, libxml2)
- Smart caching by version hash (80% cost reduction)
- Dependency bundling with manifest generation
- Weekly auto-refresh + manual triggers
- PowerShell download helper script
- Comprehensive usage documentation

Sync Workflow Fix:
- Allow .github/ commits (CI/CD config) on master
- Detect and reject code commits outside .github/
- Merge upstream while preserving .github/ changes
- Create issues only for actual pristine violations

Documentation:
- Complete Windows build usage guide
- Update all status docs to 100% complete
- Phase 3 completion summary

All three CI/CD phases complete (100%):
✅ Hourly upstream sync with .github/ preservation
✅ AI-powered PR reviews via Bedrock Claude 4.5
✅ Windows dependency builds with smart caching

Cost: $40-60/month total
See .github/PHASE3_COMPLETE.md for details

Fix sync to allow 'dev setup' commits on master

The sync workflow was failing because the 'dev setup v19' commit
modifies files outside .github/. Updated workflows to recognize
commits with messages starting with 'dev setup' as allowed on master.

Changes:
- Detect 'dev setup' commits by message pattern (case-insensitive)
- Allow merge if commits are .github/ OR dev setup OR both
- Update merge messages to reflect preserved changes
- Document pristine master policy with examples

This allows personal development environment commits (IDE configs,
debugging tools, shell aliases, Nix configs, etc.) on master without
violating the pristine mirror policy.

Future dev environment updates should start with 'dev setup' in the
commit message to be automatically recognized and preserved.

See .github/docs/pristine-master-policy.md for complete policy
See .github/DEV_SETUP_FIX.md for fix summary

Optimize CI/CD costs by skipping builds for pristine commits

Add cost optimization to Windows dependency builds to avoid expensive
builds when only pristine commits are pushed (dev setup commits or
.github/ configuration changes).

Changes:
- Add check-changes job to detect pristine-only pushes
- Skip Windows builds when all commits are dev setup or .github/ only
- Add comprehensive cost optimization documentation
- Update README with cost savings (~40% reduction)

Expected savings: ~$3-5/month on Windows builds, ~$40-47/month total
through combined optimizations.

Manual dispatch and scheduled builds always run regardless.
gburd added 2 commits July 8, 2026 10:56
Review every PR (including drafts) with two jobs that authenticate to AWS
Bedrock (Claude Opus 4.8) via GitHub OIDC (vars.AWS_ROLE_ARN); no static
AWS credentials are stored in the repo.

- ocr-review: runs Alibaba Open Code Review through an ephemeral LiteLLM
  proxy bridging OCR's OpenAI protocol to Bedrock, and posts inline review
  comments. Uses output_config.effort=xhigh (Opus 4.8 adaptive thinking).
  Path-scoped rules (.github/ocr/rule.json) encode PostgreSQL community
  review standards plus reviewer discipline (verify against the diff, don't
  hallucinate, state confidence, be blunt, accuracy over approval).
- pg-history: OCR cannot call MCP, so a separate Bedrock tool-use agent
  (.github/ocr/pg-history.py) queries the Agora MCP server (pg.ddx.io) to
  tie the change to git + pgsql-hackers history, and upserts a comment
  linking threads as https://pg.ddx.io/m/pgsql-hackers/<message-id>.
The pg-history workflow job has been failing every run with 'Bedrock call
failed: The read operation timed out' -- botocore's default 60s read
timeout on bedrock-runtime is too short for a multi-round (MAX_ROUNDS=14)
tool-use loop against a large PR diff on a reasoning-capable model; a
single converse() call alone can take several minutes under load (the
sibling ocr-review job's own LLM pass over a similarly large diff took
30-40 minutes).  Confirmed via two consecutive live runs against PR #26.

Set read_timeout=900s (15 min) explicitly via botocore.config.Config;
leave connect_timeout short since a stuck TCP handshake is a different,
cheaper-to-detect failure mode that shouldn't wait as long.
@github-actions github-actions Bot force-pushed the master branch 2 times, most recently from 40f3392 to 64f184c Compare July 8, 2026 15:12
@github-actions github-actions Bot force-pushed the master branch 9 times, most recently from 14db4ac to bad9884 Compare July 9, 2026 17:14
gburd added 8 commits July 9, 2026 16:44
A seqlock protects read-mostly, rarely-written shared data with a single
sequence counter (even = stable, odd = write in progress).  Readers take no
lock and pay no atomic read-modify-write and no StoreLoad fence on the
common path: they read the counter, copy the data into local variables,
re-read the counter, and retry if it changed.  A writer -- serialized by
the caller's own mutex -- bumps the counter odd, mutates the single copy in
place, and bumps it even.

This is cheaper than a left-right read (which needs a per-read SeqCst fence)
and far cheaper than an LWLock shared acquire (a CAS on a shared counter),
at the cost that readers may retry under a concurrent writer, so it suits
read-mostly data with short, infrequent writes.

Documented in the lmgr README and covered by a test_seqlock module that
verifies the counter transitions, the read/retry handshake, and torn-read
rejection.
Introduce two TableAmRoutine booleans and the begin_bulk_insert callback
that the UNDO subsystem builds on, plus the RelationAmSupportsUndo()
accessor index AMs use to gate UNDO record generation on the parent
table.

am_supports_undo marks an AM that registers an UNDO resource manager and
emits UNDO records tagged with its own rmid; the UNDO core stays
AM-agnostic and interprets the payload only through that RM's callbacks.
am_inplace_update_keeps_tid marks an AM that updates in place and keeps
the row's TID, so the executor can skip redundant index re-inserts for
unchanged keys.  The heap AM leaves both false.

This commit only adds the routine fields and the accessor; no AM sets the
flags yet.
Add the AM-agnostic UNDO engine: the in-WAL UNDO record format and
insertion path, the per-relation RELUNDO fork with its own resource
manager, the shared sLog tuple-state map, the rollback apply driver and
compensation-log generation, the discard horizon, and the background
revert/undo workers.  Register the UNDO, ATM, and RELUNDO resource
managers and wire the subsystem into transaction start/commit/abort,
two-phase commit, recovery, and process startup.

The engine interprets UNDO payloads only through per-RM callbacks and the
RelUndo*_hook function pointers (defined here, left NULL), so the core has
no compile-time knowledge of any specific access method.  Heap, vacuum,
pruning, reloptions, and executor integration consume only the
AM-agnostic interfaces.

No UNDO-producing AM is registered yet: RegisterUndoRmgrs() initializes
the dispatch table but registers no per-AM handlers.  The index-AM apply
handlers and the AM that sets am_supports_undo arrive in later commits.
Add the UNDO resource-manager handlers for the nbtree and hash index
AMs and register them from RegisterUndoRmgrs().  On rollback of an
aborting transaction, the nbtree handler re-descends to the leaf entry
by key and heap TID before marking it dead, so a committed entry that
shifted onto the recorded slot under concurrent inserts or leaf splits
is never killed; entries inside posting-list tuples are left for VACUUM.
The hash handler reverses its own inserts analogously.  Both are gated by
RelationAmSupportsUndo() on the parent table, so they are inert until an
UNDO-supporting table AM exists.
Add pg_setxattr/pg_getxattr/pg_removexattr/pg_listxattr wrappers over the
platform extended-attribute syscalls (Linux/*BSD/macOS, no-op stubs where
unsupported) and build them into libpgport.  The transactional file-ops
resource manager added next uses these to record and reverse xattr
mutations.
Filesystem mutations the server performs outside the buffer manager --
creating and removing directories, copying trees, writing version files,
managing symlinks, setting extended attributes -- historically had no WAL
coverage and relied on best-effort cleanup callbacks that do not survive a
crash.  Add FILEOPS: a resource manager (RM_FILEOPS_ID) that records each
mutation as a deferred pending operation, executes it at commit, and
WAL-logs it so redo reproduces it during crash recovery and standby replay.

This commit adds the operation engine, the public FileOps* API, the WAL
record formats, the redo handler, and the rmgrdesc descriptors.  A README
under src/backend/storage/file explains why FILEOPS exists and how to use
the API.  The rollback (UNDO) side and the rewiring of existing callers
follow in subsequent commits.
PostgreSQL's heap appends a new tuple version on every UPDATE and
abandons the old one to VACUUM.  For update-heavy workloads on narrow
hot rows -- counters, queue heads, running aggregates, time-series
tail writes -- the resulting version churn dominates buffer traffic,
bloats relations, and keeps autovacuum permanently behind.

RECNO is a table access method (amname "recno", RM_RECNO_ID) that
updates tuples in place.  An UPDATE overwrites the committed bytes on
the main fork; the prior image is preserved in the per-relation UNDO
fork (RELUNDO_FORKNUM) so a ROLLBACK restores it and a concurrent
snapshot reader can still reconstruct the version it is entitled to
see.  Heap is untouched; a relation opts in with USING recno.

Visibility is driven by a hybrid logical clock rather than per-tuple
xmin/xmax scans.  Each tuple header carries an HLC commit stamp; a
snapshot captures an HLC horizon and a tuple is visible when its
stamp precedes the horizon and its writer has committed.  Transient
state (uncommitted / deleted / updated) lives in header flag bits
that UNDO clears on rollback through the RelUndoClearTransientFlags
hook, and in-place delta reversal runs through RelUndoReverseDelta;
RECNO installs both at init via RecnoRelUndoInstallHooks() so the
UNDO core never sees a RECNO tuple layout.

Same-page updates rewrite the slot directly; updates that no longer
fit spill to an overflow chain with optional per-attribute dictionary
compression (LZ4/ZSTD, ANALYZE-refreshed).  A sparsemap-backed dirty
map and a partitioned secondary log (sLog) serve before-images to old
readers without a separate version-store cache; free space and
visibility are tracked in RECNO-private FSM and VM forks.

WAL coverage is complete: insert, in-place and out-of-place update,
delete, tuple-lock, VACUUM, dict writes, and overflow all log redo
records under RM_RECNO_ID with matching recovery routines, and crash
recovery cooperates with the UNDO driver to roll back in-place loser
transactions.  Index AMs over a RECNO table force a heap recheck on
index-only scans and tolerate stale index entries left by in-place
updates until before-image reclamation retires them.

The commit also wires RECNO into the supporting infrastructure:
pg_am / pg_proc catalog entries, the recno rmgr description routine,
pageinspect coverage (pageinspect 1.13 -> 1.14), the in-tree
documentation chapter, and isolation, regression, and crash-recovery
test suites exercising MVCC correctness, overflow, dictionary
compression, and dual-mode UNDO rollback.
@github-actions github-actions Bot force-pushed the master branch 2 times, most recently from 3747550 to 8d8bcc4 Compare July 10, 2026 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant